<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.6"/>
<title>IVF-HNSW: DRAFT</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/javascript">
  $(document).ready(function() { searchBox.OnSelectItem(0); });
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td style="padding-left: 0.5em;">
   <div id="projectname">IVF-HNSW
   </div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.6 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search",false,'Search');
</script>
  <div id="navrow1" class="tabs">
    <ul class="tablist">
      <li><a href="index.html"><span>Main&#160;Page</span></a></li>
      <li class="current"><a href="pages.html"><span>Related&#160;Pages</span></a></li>
      <li><a href="annotated.html"><span>Classes</span></a></li>
      <li><a href="files.html"><span>Files</span></a></li>
      <li>
        <div id="MSearchBox" class="MSearchBoxInactive">
        <span class="left">
          <img id="MSearchSelect" src="search/mag_sel.png"
               onmouseover="return searchBox.OnSearchSelectShow()"
               onmouseout="return searchBox.OnSearchSelectHide()"
               alt=""/>
          <input type="text" id="MSearchField" value="Search" accesskey="S"
               onfocus="searchBox.OnSearchFieldFocus(true)" 
               onblur="searchBox.OnSearchFieldFocus(false)" 
               onkeyup="searchBox.OnSearchFieldChange(event)"/>
          </span><span class="right">
            <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
          </span>
        </div>
      </li>
    </ul>
  </div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
<a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(0)"><span class="SelectionMark">&#160;</span>All</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(1)"><span class="SelectionMark">&#160;</span>Classes</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(2)"><span class="SelectionMark">&#160;</span>Functions</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(3)"><span class="SelectionMark">&#160;</span>Variables</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(4)"><span class="SelectionMark">&#160;</span>Typedefs</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(5)"><span class="SelectionMark">&#160;</span>Pages</a></div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">DRAFT </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h1>Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors</h1>
<p>This is the code for the paper:</p>
<p><a href="https://arxiv.org/abs/1802.02422">Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors</a>, <br/>
 Dmitry Baranchuk, Artem Babenko, Yury Malkov</p>
<p>The code is developed upon the <a href="https://github.com/facebookresearch/faiss">FAISS</a> library.</p>
<h3>Build</h3>
<p>Today we provide the C++ implementation supporting only the CPU version, which requires a BLAS library.</p>
<p>The code requires a C++ compiler that understands:</p>
<ul>
<li>the Intel intrinsics for SSE instructions</li>
<li>the GCC intrinsic for the popcount instruction</li>
<li>basic OpenMP</li>
</ul>
<h4>Installation instructions</h4>
<p>1) Clone repository ``` git clone <a href="https://github.com/dbaranchuk/ivf-hnsw">https://github.com/dbaranchuk/ivf-hnsw</a> &ndash;recursive ``` 2) Configure FAISS</p>
<p>There are a few models for makefile.inc in the faiss/example_makefiles/ subdirectory. Copy the relevant one for your system to faiss/ and adjust to your needs. In particular, for ivf-hnsw project, you need to set a proper BLAS library paths. There are also indications for specific configurations in the troubleshooting section of the <a href="https://github.com/facebookresearch/faiss/wiki/Troubleshooting">FAISS wiki</a></p>
<p>3) Replace FAISS CMakeList.txt</p>
<p>Replace faiss/CMakeList.txt with CMakeList.txt.faiss in order to deactivate building of unnecessary tests and the GPU version. ``` mv CMakeLists.txt.faiss faiss/CMakeLists.txt ```</p>
<p>4) Build project ``` cmake . make ```</p>
<h3>Data</h3>
<p>The proposed methods are tested on two 1 billion datasets: SIFT1B and DEEP1B. For using provided examples, all data files have to be in data/SIFT1B and data/DEEP1B.</p>
<h4>Data files:</h4>
<ul>
<li>SIFT1B:<ul>
<li>dataset, <a href="http://corpus-texmex.irisa.fr/">Datasets for approximate nearest neighbor search</a> ``` cd data/SIFT1B bash load_sift1b.sh ```</li>
<li>learned 993127 centroids, <a href="https://drive.google.com/file/d/1p9Aq5lTiXzmuP1ftJAIqKYEEN5EVBZsS/view?usp=sharing">GoogleDrive</a></li>
<li>precomputed indices of assigned base points, <a href="https://drive.google.com/file/d/1iFgzY2niWsCwKCPpbsjZh1urudrswEyL/view?usp=sharing">GoogleDrive</a></li>
</ul>
</li>
<li>DEEP1B:<ul>
<li>dataset, <a href="https://yadi.sk/d/11eDCm7Dsn9GA">YandexDrive</a> ``` cd data/DEEP1B python load_deep1b.py ```</li>
<li>learned 999973 centroids, <a href="https://drive.google.com/file/d/1loJ0rEIBORM34vsVSZrNeJrq1OtrcmKu/view?usp=sharing">GoogleDrive</a></li>
<li>precomputed indices of assigned base points, <a href="https://drive.google.com/file/d/10DMFnLUs5Fdr_BCht9nsa2vSyG1LKJeV/view?usp=sharing">GoogleDrive</a></li>
</ul>
</li>
</ul>
<p>Note: precomputed indices are optional, as it just lets avoid assigning step, which takes about 2-3 days for 2^20 centroids.</p>
<h3>Run</h3>
<p>tests/ provides two tests for each dataset:</p>
<ul>
<li>IVFADC</li>
<li>IVFADC + Grouping (+ Pruning)</li>
</ul>
<p>Each test requires many options, so we provide bash scripts in examples/, exploiting these tests. Scripts are commented and the <a class="el" href="structParser.html">Parser</a> class provides short descriptions for each option.</p>
<p>Make sure that:</p>
<ul>
<li>models/SIFT1B/ and models/DEEP1B/ exist ``` mkdir models ; mkdir models/SIFT1B ; mkdir models/DEEP1B ```</li>
<li>the data is placed to the data/SIFT1B/ and data/DEEP1B/ respectively (or just make symbolic links)</li>
<li>run, for example:</li>
</ul>
<p><code>bash examples/run_deep1b_grouping_OPQ.sh</code></p>
<h3>Documentation</h3>
<p>The <a href="https://cdn.rawgit.com/dbaranchuk/ivf-hnsw/fe2e4a85/docs/html/annotated.html">doxygen documentation</a> gives per-class information</p>
<h3>Citing</h3>
<p>If you find this code useful in your research, please cite:</p>
<p>{, author = {}, title = {}, booktitle = {}, year = {2018}, url = {} } </p>
</div></div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Thu Feb 8 2018 16:55:50 for IVF-HNSW by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/>
</a> 1.8.6
</small></address>
</body>
</html>
